Region specific encoding and SAO-sensitive-slice-width-adaptation for improved-quality HEVC encoding

ABSTRACT

A method provides for encoding a frame of video using an edge map made up of one or more edge-blocks detected in the frame. When the edge-blocks are contiguous, at least one slice partition is formed using the edge-blocks and the slice partition is encoded using a sample adaptive offset (SAO) filter, wherein the slice partition is formed with an adaptive slice width, and the sample adaptive offset (SAO) filter is turned on or off during the encoding based on whether the edge-blocks are being encoded. When the edge-blocks are not contiguous, edge-block processing is performed around edges in the frame during encoding of the edge-blocks. The edge-block processing involves configuring one or more of: an intra block size, a transform block size, an inter prediction block size, a quantization parameter, candidate modes for intra prediction, pyramid level for motion estimation, and fractional pixel motion estimation search.

CROSS-REFERENCE TO RELATED APPLICATION

This application claims benefit of U.S. Provisional Patent ApplicationNo. 62/362,168, filed Jul. 14, 2016, by Shailesh Ramamurthy, PadmassriChandrashekar and Anilkumar Nellore, and entitled “REGION SPECIFICENCODING AND SAO-SENSITIVE-SLICE-WIDTH-ADAPTATION FOR IMPROVED-QUALITYHEVC ENCODING,” which application is hereby incorporated by referenceherein.

BACKGROUND 1. Field of the Invention

The present invention relates to systems and methods for encoding videodata, and in particular to a system for region-specific encoding andsample adaptive offset (SAO) sensitive slice-width-adaptation forimproved-quality high efficiency video coding (HEVC) encoding.

2. Description of the Related Art

Block-based hybrid video encoding schemes such as high efficiency videocoding (HEVC) achieve compression not only by removing redundantinformation from the bitstream, but also by making adjustments in thequality of the output bitstream. While such quality compromises renderHEVC an inherently lossy process, such compromises may be made in aminimally perceptible way. The quality of the output bitstream may becontrolled by varying a number of parameters used in the encodingprocess.

Unless encoders use modeling of human visual system (HVS) todifferentiate between different parts of a scene content, the encoderquality would be typically found wanting. If the quality of a reference(which could be a reference frame in non-scalable encoding or areference frame from a previous layer for scalable coding) is notimproved using guidance from HVS, the subsequent coded portions of thebitstream would show areas of opportunity for optimization.

For example, some pixels could be part of edges or textures. Typically,quantization as decided by a naïve rate control may make regions havingtexture and/or edges suffer from a loss of detail. Coarse quantizationin general reduces high frequency information, which is important foredges or textures, and this has a ripple effect when sub-optimally codededges/textures are used as a reference in intra or inter predictions, orin scalable coding.

It is also important to be aware of moving versus static edges/texturesduring the coding process. Moving edges with compression artifacts giverise to mosquito noise, which is annoying in terms of perceptual videoquality.

Accordingly, there is a need for improved region-specific encoding. Thisneed is met by the methods and systems discussed below.

SUMMARY

To address the requirements described above, the present inventiondiscloses methods for encoding a frame of video.

In one embodiment, the method comprises detecting an edge map comprisedof one or more edge-blocks in the frame. The edge map is detected by anedge operator. The edge map is detected by classification of pixels inthe frame as edges or non-edges, and by classification of blocks asedge-blocks or non-edge-blocks based on the classification of thepixels. Specifically, the edge map is detected by a gradient ordifferences computation in a pixel domain of the frame, wherein a lowerthreshold and a higher threshold are used on the gradient or differencescomputation in order to generate the edge map, wherein the lowerthreshold and the higher threshold are used on a number of edge pixelsper individual block to classify the individual block as one of theedge-blocks or one of the non-edge-blocks, and wherein the lower andhigher threshold are scaled based on the individual block's size usedduring the encoding for decisions within the individual block.

The method also comprises, when the edge-blocks are contiguous, formingat least one slice partition using the edge-blocks and encoding theslice partition using a sample adaptive offset (SAO) filter. The slicepartition is formed with an adaptive slice width, and the sampleadaptive offset (SAO) filter is turned on or off during the encodingbased on whether the edge-blocks are being encoded.

In addition, the method comprises, when the edge-blocks are notcontiguous, performing edge-block processing around edges in the frameduring encoding of the edge-blocks. The edge-block processing involvesconfiguring one or more of: an intra block size, a transform block size,an inter prediction block size, a quantization parameter, candidatemodes for intra prediction, pyramid level for motion estimation, andfractional pixel motion estimation search.

In another embodiment, the method comprises detecting an edge map in theframe, wherein the edge map is detected using one or more edgeoperators.

The method also comprises choosing a slice width for a slice of theimage, based on the detected edge map.

In addition, the method comprises selectively turning a sample adaptiveoffset (SAO) filter on or off for the slice at the chosen slice width,based on the detected edge map.

The edge map contains one or more blocks containing edges, andedge-block processing is performed dynamically during encoding of theedge-blocks by configuring for: smaller prediction block sizes aroundedges for intra; smaller transform unit sizes around edges for intra orinter; improved quantization parameter (QP) and smaller block sizes foredge-blocks that will be used as references for intra or inter; improvedprediction by more intense sub-pixel motion estimation (ME); and motionvectors (MVs) of panned regions showing many adjacent blocks havingnearly the same motion vectors, wherein a global motion vector is usedto detect panned regions.

Still another embodiment is evidenced by an apparatus having a processorfor performing the foregoing methods.

BRIEF DESCRIPTION OF THE DRAWINGS

Referring now to the drawings in which like reference numbers representcorresponding parts throughout:

FIG. 1 is a diagram depicting an exemplary embodiment of a videocoding-decoding system that can be used for transmission and/or storageand retrieval of audio and/or video information;

FIG. 2A is a diagram of one embodiment of a codec system in which theencoded AV information is transmitted to and received at anotherlocation;

FIG. 2B is a diagram depicting an exemplary embodiment of codec systemin which the encoded information is stored and later retrieved forpresentation, hereinafter referred to as codec storage system;

FIG. 2C is another diagram depicting an exemplary content distributionsystem comprising a coding system or encoder and a decoding system ordecoder that can be used to transmit and receive HEVC data;

FIG. 3 is a block diagram illustrating one embodiment of the sourceencoder;

FIG. 4 is a diagram depicting a picture of audio visual information,such as one of the pictures in the picture sequence;

FIG. 5A is a diagram showing an exemplary partition of a coding treeunit into coding units;

FIG. 5B is a diagram showing a luma (Y), two chroma samples Cb and Cr,and associated syntax elements, used in coding tree blocks, codingblocks, prediction blocks and transform blocks;

FIG. 6 is a diagram illustrating a representation of a representativequadtree and data parameters for the code tree block partitioning shownin FIG. 5A;

FIG. 7 is a diagram illustrating the partition of a coding unit into oneor more prediction units;

FIG. 8 is a diagram showing a coding unit partitioned into fourprediction units and an associated set of transform units;

FIG. 9 is a diagram showing a residual quad tree for the transform unitsassociated with the coding unit in the example of FIG. 8;

FIG. 10 is a diagram illustrating spatial prediction;

FIG. 11 is a diagram illustrating temporal prediction;

FIG. 12 is a diagram illustrating the use of motion vector predictors;

FIGS. 13A-F are graphs of pixel level vs. pixel index;

FIGS. 14A-D illustrate different Edge Types, and how an edge is searchedacross one of the directions;

FIG. 15 is an image of a typical edge map detected by an edge operator;

FIG. 16 illustrates how a sample is classified into one of fivecategories;

FIG. 17 shows an example of the four consecutive bands that are modifiedby adding the values denoted as band offsets;

FIG. 18 illustrates the structure of slices in HEVC, wherein the slicesare groups of CTUs in scan order, separated by a slice header;

FIG. 19 illustrates the bitstream structure of HEVC that includesmultiple slices;

FIG. 20 illustrates multiple slices from FIG. 15 using adaptive slicewidths;

FIG. 21 is an example edge map, based on FIG. 20, that shows differentarrangements and orientations of edge blocks and non-edge blocks in aframe;

FIGS. 22A-B are non-limiting examples of how some embodiments of theinvention would work on processors with multicore architectures;

FIGS. 23A-B illustrate the differences in quality resulting from thisinvention;

FIG. 24 is a diagram illustrating an exemplary computer system 2200 thatcould be used to implement elements of the present invention; and

FIG. 25 is a flowchart illustrating the steps or functions performed bya processor, according to one embodiment.

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS

In the following description, reference is made to the accompanyingdrawings which form a part hereof, and which is shown, by way ofillustration, several embodiments of the present invention. It isunderstood that other embodiments may be utilized and structural changesmay be made without departing from the scope of the present invention.

HEVC SUMMARY Audio-Visual Information Transception and Storage

FIG. 1 is a diagram depicting an exemplary embodiment of a videocoding-decoding (codec) system 100 that can be used for transmissionand/or storage and retrieval of audio and/or video information. Thecodec system 100 comprises an encoding system 104, which acceptsaudio-visual (AV) information 102 (also referred to hereinafter asuncoded video) and processes the AV information 102 to generate encoded(compressed) AV information 106, and a decoding system 112, whichprocesses the encoded AV information 106 to produce recovered AVinformation 114. Since the encoding and decoding processes are notlossless, the recovered AV information 114 is not identical to theinitial AV information 102, but with judicious selection of the encodingprocesses and parameters, the differences between the recovered AVinformation 114 and the unprocessed AV information 102 are acceptable tohuman perception.

The encoded AV information 106 is typically transmitted or stored andretrieved before decoding and presentation, as performed by transception(transmission and reception) or storage/retrieval system 108.Transception losses may be significant, but storage/retrieval losses aretypically minimal or non-existent, hence, the transcepted AV information110 provided to the decoding system 112 is typically the same as orsubstantially the same as the encoded AV information 106.

FIG. 2A is a diagram of one embodiment of a codec system 200A in whichthe encoded AV information 106 is transmitted to and received at anotherlocation. A transmission segment 230 converts an input AV information102 into a signal appropriate for transmission and transmits theconverted signal over the transmission channel 212 to the receptionsegment 232. The reception segment 232 receives the transmitted signal,and converts the received signal into the recovered AV information 114for presentation. As described above, due to coding and transmissionlosses and errors, the recovered AV information 114 may be of lowerquality than the AV information 102 that was provided to thetransmission segment 230. However, error-correcting systems may beincluded to reduce or eliminate such errors. For example, the encoded AVinformation 106 may be forward error correction (FEC) encoded by addingredundant information, and such redundant information can be used toidentify and eliminate errors in the reception segment 232.

The transmission segment 230 comprises one or more source encoders 202to encode multiple sources of AV information 102. The source encoder 202encodes the AV information 102 primarily for purposes of compression toproduce the encoded AV information 106, and may include, for example aprocessor and related memory storing instructions implementing a codecsuch as MPEG-1, MPEG-2, MPEG-4 AVC/H.264, HEVC or similar codec, asdescribed further below.

The codec system 200A may also include optional elements indicated bythe dashed lines in FIG. 2A. These optional elements include a videomultiplex encoder 204, an encoding controller 208, and a videodemultiplexing decoder 218. The optional video multiplex encoder 204multiplexes encoded AV information 106 from an associated plurality ofsource encoder(s) 202 according to one or more parameters supplied bythe optional encoding controller 208. Such multiplexing is typicallyaccomplished in the time domain and is data packet based.

In one embodiment, the video multiplex encoder 204 comprises astatistical multiplexer, which combines the encoded AV information 106from a plurality of source encoders 202 so as to minimize the bandwidthrequired for transmission. This is possible, since the instantaneous bitrate of the coded AV information 106 from each source encoder 202 canvary greatly with time according to the content of the AV information102. For example, scenes having a great deal of detail and motion (e.g.sporting events) are typically encoded at higher bitrates than sceneswith little motion or detail (e.g. portrait dialog). Since each sourceencoder 202 may produce information with a high instantaneous bitratewhile another source encoder 202 produces information with a lowinstantaneous bit rate, and since the encoding controller 208 cancommand the source encoders 202 to encode the AV information 102according to certain performance parameters that affect theinstantaneous bit rate, the signals from each of the source encoders 202(each having a temporally varying instantaneous bit rate) can becombined together in an optimal way to minimize the instantaneous bitrate of the multiplexed stream 205.

As described above, the source encoder 202 and the video multiplex coder204 may optionally be controlled by an encoding controller 208 tominimize the instantaneous bit rate of the combined video signal. In oneembodiment, this is accomplished using information from a transmissionbuffer 206 which temporarily stores the coded video signal and canindicate the fullness of the buffer 206. This allows the codingperformed at the source encoder 202 or video multiplex coder 204 to be afunction of the storage remaining in the transmission buffer 206.

The transmission segment 230 also may comprise a transmission encoder210, which further encodes the video signal for transmission to thereception segment 232. Transmission encoding may include for example,the aforementioned FEC coding and/or coding into a multiplexing schemefor the transmission medium of choice. For example, if the transmissionis by satellite or terrestrial transmitters, the transmission encoder210 may encode the signal into a signal constellation beforetransmission via quadrature amplitude modulation (QAM) or similarmodulation technique. Also, if the encoded video signal is to bestreamed via an Internet protocol device and the Internet, thetransmission encodes the signal according to the appropriate protocol.Further, if the encoded signal is to be transmitted via mobiletelephony, the appropriate coding protocol is used, as further describedbelow.

The reception segment 232 comprises a transmission decoder 214 toreceive the signal that was coded by the transmission encoder 210 usinga decoding scheme complementary to the coding scheme used in thetransmission decoder 214. The decoded received signal may be temporarilystored by optional reception buffer 216, and if the received signalcomprises multiple video signals, the received signal is multiplexdecoded by video multiplex decoder 218 to extract the video signal ofinterest from the video signals multiplexed by the video multiplexencoder 204. Finally, the video signal of interest is decoded by sourcedecoder 220 (hereinafter also referred to as a target decoding device)using a decoding scheme or codec complementary to the codec used by thesource encoder 202 to encode the AV information 102.

In one embodiment, the transmitted data comprises a packetized videostream transmitted from a server (representing the transmitting segment230) to a client (representing the receiving segment 232). In this case,the transmission encoder 210 may packetize the data and embed networkabstract layer (NAL) units in network packets. NAL units define a datacontainer that has header and coded elements, and may correspond to avideo frame or other slice of video data.

The compressed data to be transmitted may packetized and transmitted viatransmission channel 212, which may include a Wide Area Network (WAN) ora Local Area Network (LAN). Such a network may comprise, for example, awireless network such as WiFi, an Ethernet network, an Internet networkor a mixed network composed of several different networks. Suchcommunication may be affected via a communication protocol, for exampleReal-time Transport Protocol (RTP), User Datagram Protocol (UDP) or anyother type of communication protocol. Different packetization methodsmay be used for each network abstract layer (NAL) unit of the bitstream.In one case, one NAL unit size is smaller than the maximum transportunit (MTU) size corresponding to the largest packet size that can betransmitted over the network without being fragmented. In this case, theNAL unit is embedded into a single network packet. In another case,multiple entire NAL units are included in a single network packet. In athird case, one NAL unit may be too large to be transmitted in a singlenetwork packet and is thus split into several fragmented NAL units witheach fragmented NAL unit being transmitted in an individual networkpacket. Fragmented NAL units are typically sent consecutively fordecoding purposes.

The reception segment 232 receives the packetized data and reconstitutesthe NAL units from the network packet. For fragmented NAL units, theclient concatenates the data from the fragmented NAL units in order toreconstruct the original NAL unit. The reception segment client 232decodes the received and reconstructed data stream and reproduces thevideo images on a display device and the audio data by a loud speaker.

FIG. 2B is a diagram depicting an exemplary embodiment of codec systemin which the encoded information is stored and later retrieved forpresentation, hereinafter referred to as codec storage system 200B. Thisembodiment may be used, for example, to locally store information in adigital video recorder (DVR), a flash drive, hard drive, or similardevice. In this embodiment, the AV information 102 is source encoded bysource encoder 202, optionally buffered by storage buffer 234 beforestorage in a storage device 236. The storage device 236 may store thevideo signal temporarily or for an extended period of time, and maycomprise a hard drive, flash drive, RAM or ROM. The stored AVinformation is then retrieved, optionally buffered by retrieve buffer238 and decoded by the source decoder 220.

FIG. 2C is another diagram depicting an exemplary content distributionsystem 200C comprising a coding system or encoder 202 and a decodingsystem or decoder 220 that can be used to transmit and receive HEVCdata.

In some embodiments, the coding system 202 can comprise an inputinterface 256, a scene change detector 249, a controller 241 a counter242 a frame memory 243, an encoding unit 244, a transmitter buffer 247and an output interface 257.

The decoding system 220 can comprise a receiver buffer 259, a decodingunit 260, a frame memory 261 and a controller 267. The coding system 202and the decoding system 220 can be coupled with each other via atransmission path which can carry a compressed bit stream. Thecontroller 241 of the coding system 202 can control the amount of datato be transmitted on the basis of the capacity of the transmitter buffer247 or receiver buffer 259 and can include other parameters such as theamount of data per a unit of time. The controller 241 can control theencoding unit 244 to prevent the occurrence of a failure of a receivedsignal decoding operation of the decoding system 220. The controller 241can be a processor or include, by way of a non-limiting example, amicrocomputer having a processor, a random access memory and a read onlymemory.

Source pictures 246 supplied from, by way of a non-limiting example, acontent provider can include a video sequence of frames including sourcepictures in a video sequence. The source pictures 246 can beuncompressed or compressed. If the source pictures 246 are uncompressed,the coding system 202 can have an encoding function. If the sourcepictures 246 are compressed, the coding system 202 can have atranscoding function. Coding units can be derived from the sourcepictures 246 utilizing the controller 241. The frame memory 243 can havea first area that can be used for storing the incoming frames from thesource pictures 246 and a second area that can be used for reading outthe frames 245 and outputting them to the encoding unit 244. Thecontroller 241 can output an area switching control signal 249 to theframe memory 243. The area switching control signal 249 can indicatewhether the first area or the second area is to be utilized.

The controller 241 can output an encoding control signal 250 to theencoding unit 244. The encoding control signal 250 can cause theencoding unit 244 to start an encoding operation, such as preparing theCoding Units of a source picture. In response to the encoding controlsignal 250 from the controller 241, the encoding unit 244 can begin toread out the prepared Coding Units to a high-efficiency encodingprocess, such as a prediction coding process or a transform codingprocess which process the prepared Coding Units generating videocompression data based on the source pictures associated with the CodingUnits.

The encoding unit 244 can package the generated video compression datain a packetized elementary stream (PES) including video packets. Theencoding unit 244 can map the video packets into an encoded video signal248 using control information and a program time stamp (PTS) and theencoded video signal 248 can be transmitted to the transmitter buffer247.

The encoded video signal 248, including the generated video compressiondata, can be stored in the transmitter buffer 247. The informationamount counter 242 can be incremented to indicate the total amount ofdata in the transmitter buffer 247. As data is retrieved and removedfrom the buffer, the counter 242 can be decremented to reflect theamount of data in the transmitter buffer 247. The occupied areainformation signal 253 can be transmitted to the counter 242 to indicatewhether data from the encoding unit 244 has been added or removed fromthe transmitter buffer 247 so the counter 242 can be incremented ordecremented. The controller 241 can control the production of videopackets produced by the encoding unit 244 on the basis of the occupiedarea information 253 which can be communicated in order to anticipate,avoid, prevent, and/or detect an overflow or underflow from taking placein the transmitter buffer 247.

The information amount counter 242 can be reset in response to a presetsignal 254 generated and output by the controller 241. After theinformation amount counter 242 is reset, it can count data output by theencoding unit 244 and obtain the amount of video compression data and/orvideo packets, which have been generated. The information amount counter242 can supply the controller 241 with an information amount signal 255representative of the obtained amount of information. The controller 241can control the encoding unit 244 so that there is no overflow at thetransmitter buffer 247.

In some embodiments, the decoding system 220 can comprise an inputinterface 266, a receiver buffer 259, a controller 267, a frame memory261, a decoding unit 260 and an output interface 268. The receiverbuffer 259 of the decoding system 220 can temporarily store thecompressed bit stream, including the received video compression data andvideo packets based on the source pictures from the source pictures 246.The decoding system 220 can read the control information andpresentation time stamp information associated with video packets in thereceived data and output a frame number signal 263 which can be appliedto the controller 267. The controller 267 can supervise the countednumber of frames at a predetermined interval. By way of a non-limitingexample, the controller 267 can supervise the counted number of frameseach time the decoding unit 260 completes a decoding operation.

In some embodiments, when the frame number signal 263 indicates thereceiver buffer 259 is at a predetermined capacity, the controller 267can output a decoding start signal 264 to the decoding unit 260. Whenthe frame number signal 263 indicates the receiver buffer 259 is at lessthan a predetermined capacity, the controller 267 can wait for theoccurrence of a situation in which the counted number of frames becomesequal to the predetermined amount. The controller 267 can output thedecoding start signal 264 when the situation occurs. By way of anon-limiting example, the controller 267 can output the decoding startsignal 264 when the frame number signal 263 indicates the receiverbuffer 259 is at the predetermined capacity. The encoded video packetsand video compression data can be decoded in a monotonic order (i.e.,increasing or decreasing) based on presentation time stamps associatedwith the encoded video packets.

In response to the decoding start signal 264, the decoding unit 260 candecode data amounting to one picture associated with a frame andcompressed video data associated with the picture associated with videopackets from the receiver buffer 259. The decoding unit 260 can write adecoded video signal 162 into the frame memory 261. The frame memory 261can have a first area into which the decoded video signal is written,and a second area used for reading out decoded pictures 262 to theoutput interface 268.

In various embodiments, the coding system 202 can be incorporated orotherwise associated with a transcoder or an encoding apparatus at aheadend and the decoding system 220 can be incorporated or otherwiseassociated with a downstream device, such as a mobile device, a set topbox or a transcoder.

Source Encoding/Decoding

As described above, the encoders 202 employ compression algorithms togenerate bit streams and/or files of smaller size than the originalvideo sequences in the AV information 102. Such compression is madepossible by reducing spatial and temporal redundancies in the originalsequences.

Encoders 202 include those compliant with the video compression standardH.264/MPEG-4 AVC (“Advanced Video Coding”) developed by between the“Video Coding Expert Group” (VCEG) of the ITU and the “Moving PictureExperts Group” (MPEG) of the ISO, in particular in the form of thepublication “Advanced Video Coding for Generic Audiovisual Services”(March 2005), which is hereby incorporated by reference herein.

HEVC “High Efficiency Video Coding” (sometimes known as H.265) isexpected to replace the H.264/MPEG-4 AVC. HEVC introduces new codingtools and entities that are generalizations of the coding entitiesdefined in H.264/AVC, as further described below.

FIG. 3 is a block diagram illustrating one embodiment of the sourceencoder 202. The source encoder 202 accepts AV information 102 and usessampler 302 to sample the AV information 102 to produce a sequence 303of successive of digital images or pictures, each having a plurality ofpixels. A picture can comprise a frame or a field, wherein a frame is acomplete image captured during a known time interval, and a field is theset of odd-numbered or even-numbered scanning lines composing a partialimage.

The sampler 302 produces a digitized (and as yet, uncompressed) picturesequence 303. Each digital picture can be represented by one or morematrices having a plurality of coefficients that represent informationabout the pixels that together comprise the picture. The value of apixel can correspond to luminance or other information. In the casewhere several components are associated with each pixel, for example,red-green-blue (RGB) or luminance-chrominance (YCbCr, wherein Y is theluma component and Cb and Cr are the blue-difference and red-differencechroma components, respectively), each of these components may beseparately processed.

Images can be segmented or partitioned into “slices,” which may comprisea portion of the picture or may comprise the entire picture. In theH.264 standard, these slices are divided into coding entities calledmacroblocks (generally blocks of size 16 pixels×16 pixels) and eachmacroblock may in turn be divided into different sizes of data blocks,for example 4×4, 4×8, 8×4, 8×8, 8×16, 16×8. HEVC expands and generalizesthe notion of the coding entity beyond that of the macroblock.

HEVC Coding Entities: CTU, CU, CTB, CB, PU and TU

Like other video coding standards, HEVC is a block-based hybrid spatialand temporal predictive coding scheme. However, HEVC introduces newcoding entities that are not included with H.264/AVC standard. Thesecoding entities include (1) coding tree units (CTUs), coding units(CUs), coding tree blocks (CTBs), coding blocks (CBs), predictive units(PUs) and transform units (TUs), and are further described below.

FIG. 4 is a diagram depicting a picture 400 of AV information 102, suchas one of the pictures in the picture sequence 303. The picture 400 isspatially divided into non-overlapping square blocks known as codingtree unit(s), or CTUs 402. Unlike H.264 and previous video codingstandards where the basic coding unit is macroblock of 16×16 pixels, theCTU 402 is the basic coding unit of HEVC, and can be as large as 128×128pixels. As shown in FIG. 4, the CTUs 402 are typically referenced withinthe picture 400 in an order analogous to a progressive scan.

Each CTU 402 may in turn be iteratively divided into smaller variablesize coding units (CUs) described by a “quadtree” decomposition furtherdescribed below. Coding units are regions formed in the image to whichsimilar encoding parameters are applied and transmitted in the bitstream314.

FIG. 5A is a diagram showing an exemplary partition of a CTU 402 intoCUs such as coding unit 502A and 502B (hereinafter alternativelyreferred to as coding unit(s) 502). A single CTU 402 can be divided intofour CUs 502 such as CU 502A, each a quarter of the size of CTU 402.Each such divided CU 502A can be further divided into four smaller CUs502B of quarter size of initial CU 502A.

The division of CTUs 402 into CUs 502A and into smaller CUs 502B isdescribed by “quadtree” data parameters (e.g. flags or bits) that areencoded into the output bitstream 314 along with the encoded data asoverhead known as syntax.

FIG. 5B is a diagram showing a luma (Y) 504, two chroma samples Cb 506and Cr 508, and associated syntax elements 510, that comprise codingtree blocks (CTBs) or coding blocks (CBs), respectively, for the CTUs402 or CUs 502A, 502B. This structure is also used for prediction blocks(PBs) associated with prediction units (PUs) and transform blocks (TBs)associated with transform units (TUs), as described in more detailbelow.

FIG. 6 is a diagram illustrating a representation of a representativequadtree 600 and data parameters for the CTU 402 partitioning shown inFIG. 5A. The quadtree 600 comprises a plurality of nodes including firstnode 602A at one hierarchical level and second node 602B at a lowerhierarchical level (hereinafter, quadtree nodes may be alternativelyreferred to as “nodes” 602). At each node 602 of a quadtree, a “splitflag” or bit “1” is assigned if the node 602 is further split intosub-nodes, otherwise a bit “0” is assigned.

For example, the CTU 402 partition illustrated in FIG. 5A can berepresented by the quadtree 600 presented in FIG. 6, which includes asplit flag of “1” associated with node 602A at the top CU 502 level(indicating there are 4 additional nodes at a lower hierarchical level).The illustrated quadtree 600 also includes a split flag of “1”associated with node 602B at the mid CU 502 level to indicate that thisCU is also partitioned into four further CUs 502 at the next (bottom) CUlevel. The source encoder 202 may restrict the minimum and maximum CU502 sizes, thus changing the maximum possible depth of the CU 502splitting.

The source encoder 202 generates encoded AV information 106 in the formof a bitstream 314 that includes a first portion having encoded data forthe CUs 502 and a second portion that includes overhead known as syntaxelements. The encoded data includes data corresponding to the encodedCUs 502 (i.e., the encoded residuals together with their associatedmotion vectors, predictors, or related residuals as described furtherbelow). The second portion includes syntax elements that may representencoding parameters which do not directly correspond to the encoded dataof the blocks. For example, the syntax elements may comprise an addressand identification of the CU 502 in the image, a quantization parameter,an indication of the elected Inter/Intra coding mode, the quadtree 600or other information.

CUs 502 correspond to elementary coding elements and include two relatedsub-units: prediction units (PUs) and transform units (TUs), both ofwhich have a maximum size equal to the size of the corresponding CU 502.Each PU and TU is comprised of prediction blocks (PBs) and transformblocks (TBs), respectively, formatted as shown in FIG. 5B, with a luma(Y) 504, two chroma samples Cb 506 and Cr 508, and associated syntaxelements 510.

FIG. 7 is a diagram illustrating the partition of a CU 502 into one ormore PUs 702. A PU 702 corresponds to a partitioned CU 502 and is usedto predict pixel values for intra-picture or inter-picture types. PUs702 are an extension of the partitioning of H.264/AVC for motionestimation, and are defined for each CU 502 that is not furthersubdivided into other CUs (“split flag”=0). At each leaf 604 of thequadtree 600, a final (bottom level) CU 502 of 2N×2N can possess one offour possible patterns of PUs: 2N×2N (702A), 2N×N (702B), N×2N (702C)and N×N (702D), as shown in FIG. 7, as well as certain other asymmetricmotion partitions (AMP) defined in the HEVC specification.

A CU 502 can be either spatially or temporally predictive coded. If a CU502 is coded in “intra” mode, each PU 702 of the CU 502 can have its ownspatial prediction direction and image information as further describedbelow. Also, in the “intra” mode, the PU 702 of the CU 502 may depend onanother CU 502 because it may use a spatial neighbor, which is inanother CU. If a CU 502 is coded in “inter” mode, each PU 702 of the CU502 can have its own motion vector(s) and associated referencepicture(s).

FIG. 8 is a diagram showing a CU 502 partitioned into four PUs 702 andan associated set of transform units (TUs) 802. TUs 802 are used torepresent the elementary units that are spatially transformed by atransform such as the DCT (Discrete Cosine Transform) or the DST(discrete sine transform). The size and location of each block transformTU 802 within a CU 502 is described by a “residual” quadtree (RQT).

FIG. 9 is a diagram showing RQT 900 for TUs 802 for the CU 502 in theexample of FIG. 8. Note that the “1” at the first node 902A of the RQT900 indicates that there are four branches and that the “1” at thesecond node 902B at the adjacent lower hierarchical level indicates thatthe indicated node further has four branches. The data describing theRQT 900 is also coded and transmitted as an overhead in the bitstream314.

The coding parameters of a video sequence may be stored in dedicated NALunits called parameter sets, which include a Video Parameter Sets (VPS)that describes the overall characteristics of coded video sequences; aSequence Parameter Set (SPS) that contains information that applies toall slices of a video sequence and is fixed within a sequence; and aPicture Parameter Set (PPS) that conveys information that could changefrom picture to picture.

Spatial and Temporal Prediction

One of the techniques used to compress a bitstream 314 is to forego thestorage of pixel values themselves and instead, predict the pixel valuesusing a process that can be repeated at the decoder 220 and store ortransmit the difference between the predicted pixel values and theactual pixel values (known as the residual). So long as the decoder 220can compute the same predicted pixel values from the informationprovided, the actual picture values can be recovered by adding theresiduals to the predicted values. The same technique can be used tocompress other data as well.

Referring back to FIG. 3, each PU 702 of the CU 502 being processed isprovided to a predictor module 307. The predictor module 307 predictsthe values of the PUs 702 based on information in nearby PUs 702 in thesame frame (intra-frame prediction, which is performed by the spatialpredictor 324) and information of PUs 702 in temporally proximate frames(inter-frame prediction, which is performed by the temporal predictor330). Temporal prediction, however, may not always be based on acollocated PU, since collocated PUs are defined to be located at areference/non-reference frame having the same x and y coordinates as thecurrent PU 702. These techniques take advantage of spatial and temporaldependencies between PUs 702.

Encoded units can therefore be categorized to include two types: (1)non-temporally predicted units and (2) temporally predicted units.Non-temporally predicted units are predicted using the current frame,including adjacent or nearby PUs 702 within the frame (e.g. intra-frameprediction), and are generated by the spatial predictor 324. Temporallypredicted units are predicted from one temporal picture (e.g. P-frames)or predicted from at least two reference pictures temporally aheadand/or behind (i.e. B-frames).

Spatial Prediction

FIG. 10 is a diagram illustrating spatial prediction of PUs 702. Apicture may comprise a PU 702 and spatially proximate other PUs 1-4,including nearby PU 702N. The spatial predictor 324 predicts the currentblock (e.g. block C of FIG. 10) by means of an “intra-frame” predictionwhich uses PUs 702 of already-encoded other blocks of pixels of thecurrent image.

The spatial predictor 324 locates a nearby PU (e.g. PU 1, 2, 3, 4 or 5of FIG. 10) that is appropriate for spatial coding and determines anangular prediction direction to that nearby PU. In HEVC, 35 directionscan be considered, so each PU may have one of 35 directions associatedwith it, including horizontal, vertical, 45 degree diagonal, 135 degreediagonal, etc. The spatial prediction direction of the PU is indicatedin the syntax.

Referring back to the spatial predictor 324 of FIG. 3, this locatednearby PU is used to compute a residual PU 704 (e) as the differencebetween the pixels of the nearby PU 702N and the current PU 702, usingelement 305. The result is an intra-predicted PU element 1006 thatcomprises a prediction direction 1002 and the intra-predicted residualPU 1004. The prediction direction 1002 may be coded by inferring thedirection from spatially proximate PUs, and the spatial dependencies ofthe picture, enabling the coding rate of the intra prediction directionmode to be reduced.

Temporal Prediction

FIG. 11 is a diagram illustrating temporal prediction. Temporalprediction considers information from temporally neighboring pictures orframes, such as the previous picture, picture i−1.

Generally, temporal prediction includes single-prediction (P-type),which predicts the PU 702 by referring to one reference area from onlyone reference picture, and multiple prediction (B-type), which predictsthe PU by referring to two reference areas from one or two referencepictures. Reference images are images in the video sequence that havealready been coded and then reconstructed (by decoding).

The temporal predictor 330 identifies, in one or several of thesereference areas (one for P-type or several for B-type), areas of pixelsin a temporally nearby frame so that they can be used as predictors ofthis current PU 702. In the case where several areas predictors are used(B-type), they may be merged to generate one single prediction. Thereference area 1102 is identified in the reference frame by a motionvector (MV) 1104 that defines the displacement between the current PU702 in current frame (picture i) and the reference area 1102 identifiedby a reference index (refIdx) in the reference frame (picture i−1). A PUin a B-picture may have up to two MVs. Both MV and refIdx informationare included in the syntax of the HEVC bitstream.

Referring again to FIG. 3, a difference between the pixel values betweenof the reference area 1102 and the current PU 702 may be computed byelement 305 as selected by switch 306. This difference is referred to asthe residual of the inter-predicted PU 1006. At the end of the temporalor inter-frame prediction process, the current PU 1006 is composed ofone motion vector MV 1104 and a residual 1106.

However, as described above, one technique for compressing data is togenerate predicted values for the data using means repeatable by thedecoder 220, computing the difference between the predicted and actualvalues of the data (the residual) and transmitting the residual fordecoding. So long as the decoder 220 can reproduce the predicted values,the residual values can be used to determine the actual values.

This technique can be applied to the MVs 1104 used in temporalprediction by generating a prediction of the MV 1104, computing adifference between the actual MV 1104 and the predicted MV 1104 (aresidual) and transmitting the MV residual in the bitstream 314. So longas the decoder 220 can reproduce the predicted MV 1104, the actual MV1104 can be computed from the residual. HEVC computes a predicted MV foreach PU 702 using the spatial correlation of movement between nearby PUs702.

FIG. 12 is a diagram illustrating the use of motion vector predictors(MVPs) in HEVC. Motion vector predictors V₁, V₂ and V₃ are taken fromthe MVs 1104 of a plurality of blocks 1, 2, and 3 situated nearby oradjacent the block to encode (C). As these vectors refer to motionvectors of spatially neighboring blocks within the same temporal frameand can be used to predict the motion vector of the block to encode,these vectors are known as spatial motion predictors.

FIG. 12 also illustrates temporal motion vector predictor V_(T) which isthe motion vector of the co-located block C′ in a previously decodedpicture (in decoding order) of the sequence (e.g. block of picture i−1located at the same spatial position as the block being coded (block Cof image i).

The components of the spatial motion vector predictors V₁, V₂ and V₃ andthe temporal motion vector predictor V_(T) can be used to generate amedian motion vector predictor V_(M). In HEVC, the three spatial motionvector predictors may be taken as shown in FIG. 12, that is, from theblock situated to the left of the block to encode (V₁), the blocksituated above (V₃) and from one of the blocks situated at therespective corners of the block to encode (V₂), according to apredetermined rule of availability. This MV predictor selectiontechnique is known as Advanced Motion Vector Prediction (AMVP).

A plurality of (typically five) MV predictor (MVP) candidates havingspatial predictors (e.g. V₁, V₂ and V₃) and temporal predictor(s) V_(T)is therefore obtained. In order to reduce the overhead of signaling themotion vector predictor in the bitstream, the set of motion vectorpredictors may be reduced by eliminating data for duplicated motionvectors (for example, MVs which have the same value as other MVs may beeliminated from the candidates).

The encoder 202 may select a “best” motion vector predictor from amongthe candidates, and compute a motion vector predictor residual as adifference between the selected motion vector predictor and the actualmotion vector, and transmit the motion vector predictor residual in thebitstream 314. To perform this operation, the actual motion vector mustbe stored for later use by the decoder 220 (although it is nottransmitted in the bit stream 314. Signaling bits or flags are includedin the bitstream 314 to specify which MV residual was computed from thenormalized motion vector predictor, and are later used by the decoder torecover the motion vector.

The intra-predicted residuals 1004 and the inter-predicted residuals1106 obtained from the spatial (intra) or temporal (inter) predictionprocess are then transformed by transform module 308 (depicted in FIG.3) into the transform units (TUs) 802 described above. A TU 802 can befurther split into smaller TUs using the RQT decomposition describedabove with respect to FIG. 9. In HEVC, generally 2 or 3 levels ofdecompositions are used and authorized transform sizes are from 32×32,16×16, 8×8 and 4×4. As described above, the transform is derivedaccording to a discrete cosine transform (DCT) or discrete sinetransform (DST).

The residual transformed coefficients are then quantized by quantizer310. Quantization plays a very important role in data compression. InHEVC, quantization converts the high precision transform coefficientsinto a finite number of possible values. Although the quantizationpermits a great deal of compression, quantization is a lossy operation,and the loss by quantization cannot be recovered.

The coefficients of the quantized transformed residual are then coded bymeans of an entropy coder 312 and then inserted into the compressed bitstream 314 as a part of the useful data coding the images of the AVinformation. Coding syntax elements may also be coded using spatialdependencies between syntax elements to increase the coding efficiency.HEVC offers entropy coding such as context-adaptive binary arithmeticcoding (CABAC). Other forms or entropy or arithmetic coding may also beused.

In order to calculate the predictors used above, the encoder 202 decodesalready encoded PUs 702 using “decoding” loop 315, which includeselements 316, 318, 320, 322, 328. This decoding loop 315 reconstructsthe PUs and images from the quantized transformed residuals.

The quantized transform residual coefficients E are provided todequantizer 316, which applies the inverse operation to that ofquantizer 310 to produce dequantized transform coefficients of theresidual PU (E′) 708. The dequantized data 708 is then provided toinverse transformer 318 which applies the inverse of the transformapplied by the transform module 308 to generate reconstructed residualcoefficients of the PU (e′) 710.

The reconstructed coefficients of the residual PU 710 are then added tothe corresponding coefficients of the corresponding predicted PU (x′)702′ selected from the intra-predicted PU 1004 and the inter-predictedPU 1106 by selector 306. For example, if the reconstructed residualcomes from the “intra” coding process of the spatial predictor 324, the“intra” predictor (x′) is added to this residual in order to recover areconstructed PU (x″) 712 corresponding to the original PU 702 modifiedby the losses resulting from a transformation, for example in this casethe quantization operations. If the residual 710 comes from an “inter”coding process of the temporal predictor 330, the areas pointed to bythe current motion vectors (these areas belong to the reference imagesstored in reference buffer 328 referred by the current image indices)are merged then added to this decoded residual. In this way the originalPU 702 is modified by the losses resulting from the quantizationoperations.

To the extent that the encoder 202 uses motion vector predictiontechniques analogous to the image prediction techniques described above,the motion vector may be stored using motion vector buffer 329 for usein temporally subsequent frames. A flag may be set and transferred inthe syntax to indicate that the motion vector for the currently decodedframe should be used for at least the subsequently coded frame insteadof replacing the contents of the MV buffer 329 with the MV for thecurrent frame.

A loop filter 322 is applied to the reconstructed signal (x″) 712 inorder to reduce the effects created by heavy quantization of theresiduals obtained, and to improve the signal quality. The loop filter322 sequentially applies a deblocking filter (DBF) and a sample adaptiveoffset (SAO) filter in the inter-picture prediction loop.

The loop filter 322 applies the DBF for smoothing borders between PUs tovisually attenuate high frequencies created by the coding process and alinear filter that is applied after all of the PUs for an image havebeen decoded to minimize the sum of the square difference (SSD) with theoriginal image. The linear filtering process is performed on a frame byframe basis and uses several pixels around the pixel to be filtered, andalso uses spatial dependencies between pixels of the frame. The linearfilter coefficients may be coded and transmitted in one header of thebitstream, typically a picture or slice header.

The loop filter 322 applies the SAO filter to allow for betterreconstruction of the original signal amplitudes by applying offsetsstored in a lookup table in the bitstream. The SAO filter can bedisabled or applied in one of two modes per CTB or CB: edge offset modeor band offset mode.

The edge offset mode operates by comparing the value of a sample to twoof its eight neighbors using one of four directional gradient patterns.Based on a comparison with these two neighbors, the sample is classifiedinto one of five categories: minimum, maximum, an edge with the samplehaving the lower value, an edge with the sample having the higher value,or monotonic. For each of the first four categories an offset isapplied.

The band offset mode applies an offset based on the amplitude of asingle sample. A sample is categorized by its amplitude into one of 32bands (histogram bins). Offsets are specified for four consecutive ofthe 32 bands, because in flat areas which are prone to bandingartifacts, sample amplitudes tend to be clustered in a small range. TheSAO filter was designed to increase picture quality, reduce bandingartifacts, and reduce ringing artifacts.

The filtered images, also known as reconstructed images, are then storedas reference images from reference image buffer 328 in order to allowthe subsequent “inter” predictions taking place during the compressionof the subsequent images of the current video sequence.

Picture Level Quantization Parameter Rate Control

For quantization, HEVC uses essentially the same uniform-reconstructionquantization (URQ) scheme controlled by a quantization parameter (QP) asin H.264/MPEG-4 AVC. The range of the QP values is defined from 0 to 51,and an increase by 6 doubles the quantization step size, such that themapping of QP values to step sizes is approximately logarithmic.Quantization scaling matrices are also supported.

To reduce the memory needed to store frequency-specific scaling values,only quantization matrices of sizes 4×4 and 8×8 are used. For the largertransformations of 16×16 and 32×32 sizes, an 8×8 scaling matrix is sentand is applied by sharing values within 2×2 and 4×4 coefficient groupsin frequency sub-spaces—except for values at DC positions, for whichdistinct values are sent and applied.

It is also advantageous to adapt QP estimates on a sub-picture basis.This can exploit the non-uniform nature of HEVC coding. Spatial maskingand distortion is less noticeable in busy (higher complexity) areas thanin non-busy (smooth or less complex) areas. Accordingly, a higher QPparameter may be assigned to a busy area of the picture (thus requiringfewer bits to code) and a lower QP parameter (resulting in encodingusing a greater number of bits) may be used in smooth areas.

Region Specific Encoding and Sao-Sensitive Slice-Width-Adaptation

In this invention, the HEVC encoder uses guidance from the HVS, or edgeor ROI (region of interest) aspects, to perform the following steps orfunctions:

-   -   1. An edge map comprised of one or more edge blocks is detected        in an image using, as a non-limiting example, one or more        hardware-assisted primitives, such as INTEL™ IPPs (Integrated        Performance Primitives) edge operators. Example edge operators        include Prewitt, Sobel, Canny, Difference of Gaussian, Laplacian        or other edge operators. Hence, the use of hardware-assisted        primitives makes for low complexity implementations, and the        complexity for this step can be expediently bounded.    -   2. Based on the edge map, a slice width may be chosen.    -   3. SAO filtering is selectively turned on or off in the loop        filter 322, based on the presence of the edge map or otherwise.        A measure of how many edge pixels are present in a specific        region can be used since, typically, regions would contain a few        or many edges (rather than none or a completely-edge-region).        More advanced fuzzy classification methods can be used. Adaptive        slice width selection is used so that the SAO filtering is        turned on only within ROI slices.

For the blocks containing edges that undergo edge based processing, thefollowing steps or functions can be configured dynamically duringencoding of such blocks:

-   -   Smaller prediction block sizes around edges (for example, 4×4 or        8×8) for intra.    -   Smaller TU sizes around edges (for example, 4×4 or 8×8) for        intra or inter.    -   Finer QP and small block-sizes for edge blocks that will be used        as references.

It can be determined as to which blocks are used as references forintra. For inter, Pass 1 encoding shows which blocks would be used asreferences.

-   -   Improve prediction by more intense sub-pixel motion estimation        (ME) (include more references).    -   Motion vectors of panned regions show many adjacent blocks        having nearly same MV (they move together in panning). MVs of        blocks which suffer more mosquito noise during encoding would be        showing different orientations. The global motion vector can be        used to detect panned regions.

The table set forth below describes how these cases can be categorizedas bins and processed accordingly.

TABLE 1 Categorization of edge maps and processing involved under thisinvention Invention uses the Category Characteristic followingprocessing Bin 1 When edge blocks are Induce a slice partition on thecontiguous, and enough in contiguous blocks that have number, form aslice partition edges and encode such slices with an adaptive slicewidth. turning SAO on. (Use empirical thresholds for Advantage: bitratesavings as the above determination.) well as computational savingscompared to unconditionally turning SAO on everywhere or turning SAO offeverywhere. Bin 2 When edge blocks are not Do edge block processingcontiguous, perform edge with maximally intense block processing. Theedge processing (e.g. around blocks are not static, and also edges): donot “move together” (this induce small intra can be detected by motionblocks (4 × 4, 8 × 8), field or motion vectors small transform blockcharacteristics). Rather, the size (4 × 4, 8 × 8), blocks move indifferent small inter prediction directions and have high edge blocksize (4 × 4, 8 × 8, content, whereupon higher asymmetric e.g. 4 × 8,mosquito noise results, i.e., 8 × 4, etc.), unlike a large “rigid bodyfiner quantization motion.” parameters (<35) independently parameterizedfor exact value for each of these intra/inter prediction block sizes,and for intra, more candidate modes, perform Motion Estimation at fullresolution as opposed to subsampled space in pyramid, heavy duty searchfor fractional pixel motion estimation (e.g. half and quarter pixel).Bin 3 Medium or low number of Progressively decrease the edges,non-contiguous. amount of processing (corresponding to medium to lownumber of edges) which has been described for bin 2. For example, use:medium intra block size (16 × 16, 32 × 32), medium transform block size(e.g. 16 × 16), medium inter prediction block size (16 × 16),Intra-relatively candidate modes than in bin 2, Motion Estimation notnecessarily at full resolution and may include direct search insub-sampled space in pyramid, lesser search in fractional pixel motionestimation (e.g. half and quarter pixel). Bin 4 No edge content or edgesNone of the above processing. “move together” as in a rigid body motionwhich does not pose mosquito noise risks.

Edge Processing

Consider the edge map as detected by a Sobel or Prewitt or Cannyoperator, for example. This is obtained by running edge detection oneach frame, which basically gives a map of gradients or adjacent-pixels,evaluated on a per-pixel-position.

Setting Gradient or Difference Limits

In order to run edge detection, the first level of parameterization isto set lower and higher limits of adjacent-pixel-differences (orgradients). A band-pass filtering of gradients (with upper and lowerthresholds parameters configured) determines which pixels are classifiedas edges.

For example, pixels may be classified as edges when the gradient fallsbetween the values of DLL (Difference-Lower-Limit) and DHL(Difference-Higher-Limit). An example for values of DLL and DHL can be80 and 255. All pixels with a gradient between DLL and DHL which areadjacent to a pixel already determined as edge also are declared as edgepixels.

Classification of Coding Block as Edge Block

In one embodiment, the classification of a coding block (CB) as an edgeblock is performed by the following steps or functions:

-   -   1. Determine the edge map using the chosen operator, as        described above.    -   2. Perform the band-pass filtering by setting gradient or        difference limits, as described above.    -   3. As an example, for a block (CU) size of 32×32, parameterize        an upper and lower thresholds parameters to classify the block        (CU) as an edge block if the number of contained pixel-edges        (obtained by step #2 above) is between these limits. In one        example, BL32 (Block-Low-edge-count-parameter for block size        32×32) and BH32 (Block-High-edge-count-parameter for block size        32×32) parameters are set to 64 and 512 for a block (CU) size of        32×32.    -   4. For edge-based processing in HEVC encoding, note that the        edge detection is carried out on a specific block size, for        example, 32×32 or 16×16, in a preprocessing step (i.e. prior to        encoding) that works on the pixel domain. The BL32 and BH32 are        directly used to make decisions on which blocks are edges.        During actual encoding, the CU sizes are decided. At this stage,        the decisions on which of the blocks were edges in the        preprocessing step can easily be used to map the CU region into        constituent edge or non-edge blocks. In other words, the CU        region contains blocks that have been determined as edge or        non-edge blocks in the preprocessing step. These edge or        non-edge blocks guide the exemplary actions as in step #5 below.        The constituent edge or non-edge blocks can be used to guide the        prediction block size and transform block size that can be used        as candidates.    -   5. For the block size under consideration, which is used for        each of the following decisions in HEVC, use the blocks that get        classified as edge-blocks from steps #3 and #4 above, to further        perform the following exemplary (non-limiting) actions:        -   induce small intra blocks (4×4, 8×8) within the block (CU)            being coded;        -   small transform block size (4×4, 8×8) within the block (CU)            being coded;        -   small inter prediction block size (4×4, 8×8, asymmetric e.g.            4×8, 8×4, etc., within the block (CU) being coded;        -   improved (finer) quantization parameters (<35) independently            parameterized for exact value for each of these intra/inter            prediction block sizes, within the block (CU) being coded;        -   for intra, more candidate modes, within the block (CU) being            coded;        -   perform Motion Estimation at full resolution as opposed to            subsampled space in pyramid, heavy duty search for            fractional pixel motion estimation (e.g. half and quarter            pixel), within the block (CU) being coded.    -   6. From experimental results, it seems better to also have a        mask which filters out only edges which are associated with        moving portions of the frame. Note that non-moving portions are        associated with static edges inherent in the static visual        content. There are several ways to determine moving edges and        create the mask. These range from simple differences between        frames to more sophisticated motion vector analysis used for        deriving the mask.

Adaptive Slice Width and Selectively Enabling the SAO Filter

Quantization makes reconstructed and original blocks differ. Thequantization error is not uniformly distributed among pixels. There is abias in distortion around edges (due to the Gibbs effect).

As noted above, HEVC uses two filtering stages in its in-loop filtering322, namely, DBF and SAO filters. The SAO filtering adds an offset to a(deblocked) pixel value according to an SAO type which is based on edgedirection/shape (edge offset) and pixel value (band offset) or unchanged(off).

In addition to correction at local extremes, HEVC allows alternativecorrection to specific ranges of pixel values.

For example, FIGS. 13A-13F are graphs of pixel level vs. pixel index(x−1, x, x+1) that illustrate the offsets in SAO filtering, whereinFIGS. 13A, 13B and 13C are graphs of positive offsets applied in thecase of local minima (local min) and FIGS. 13D, 13E and 13F are graphsof negative offsets applied in case of local maxima (local max). Thearrows in each graph at pixel index x show the offset direction.

It has been reported that SAO filtering reduces ringing and mosquitoartifacts (which become more serious with large transforms) and improvessubjective quality for low compression ratio video.

SAO filtering can be optionally turned off or applied only on lumasamples or only on chroma samples (regulated by slice_sao_luma_flag andslice_sao_chroma_flag). SAO parameters can be either explicitly signaledin the CTU header or inherited from left or above CTUs.

As mentioned above, there are two types of SAO filtering:

-   -   Edge Type—offset depends on edge mode (signaled by        SaoTypeIdx=2); and    -   Band Type—offset depends on the sample amplitude (SaoTypeIdx=1).

In the case of Edge Type, the edge is searched across one of thedirections signaled by the value of the sao_eo_class parameter (=0, 1,2, 3), as shown in FIGS. 14A-D, once per CTU 1400, 1402, 1404, 1406,wherein the sample labeled “p” indicates a current sample and the twosamples labeled “n0” and “n1” specify two neighboring samples along thechosen direction.

The detected edge can come from an edge map using an edge detectionoperator (Canny edge detector operator is a non-limiting example). Atypical edge map detected by an edge operator is shown in FIG. 15.

The edge detection is applied to each sample. According to the results,the sample is classified into five categories (EdgeIdx=0, 1, 2, 3, 4),as shown in FIG. 16, wherein each EdgeIdx has a corresponding conditionwith regard to the samples labeled “p”,“n0” and “n1”, and an associatedmeaning. For the EdgeIdx=0, the condition comprises p=n0 and p=n1, whichhas the meaning of “flat area”; for the EdgeIdx=1, the conditioncomprises p<n0 and p<n1, which has the meaning of “local min”; for theEdgeIdx=2, the condition comprises p<n0 and p=n1 or p<n1 and p=n0, whichhas the meaning of “edge”; for the EdgeIdx=3, the condition comprisesp>n0 and p=n1 or p>n1 and p=n0, which has the meaning of “edge”; and forthe EdgeIdx=4, the condition comprises p>n0 and p>n1, which has themeaning of “local max”.

According to EdgeIdx, the corresponding sample offset (signaled bysao_offset_abs and sao_offset_sign) is added to the current sample. Upto 12 edge offsets (4 luma, 4 Cb chroma and 4 Cr chroma) are signaledper CTU.

Under the band offset method, the pixel range from 0 to 255 (for 8 bitdepth) is uniformly split into 32 bands and the sample values belongingto four consecutive bands are modified by adding the values denoted asband offsets. FIG. 17 shows an example of the pixel range 1700 from 0 to255 where four consecutive bands 1702 are modified by adding the valuesdenoted as band offsets. The band offsets are signaled in the CTUheader. Experimental results reveal that Band Type SAO is beneficial innoisy sequences or in sequences with large gradients.

Slices in HEVC

FIG. 18 illustrates the structure 1800 of slices in HEVC, wherein theslices are groups of CTUs in scan order, separated by a start code(slice header). In this example, five slice partitions are shown,labeled as Slice #0 (1802), #1 (1804), #2 (1806), #3 (1808) and #4(1810).

Generally, slices are used for:

-   -   network packetization (MTU size matching);    -   parallel processing (slices are self-contained, excepting        deblocking), but the decoder has to perform some preprocessing        to identify entry points; and    -   fast resynchronization in case of bitstream errors or packet        loss.

FIG. 19 illustrates the bitstream structure 1900 of HEVC, wherein theNAL unit of the bitstream includes a VPS (1902), SPS (1904), and PPS(1906), followed by three instances of slice headers (1908) and slicedata (1910) for Pictures #1, #2, #3. In this structure the SPS (1904)points at the PPS (1906), and the PPS (1906) points at each slice header(1908).

A slice comprises an integral number of CTBs or macroblocks. The numberof CTBs in a slice is typically configured as a fixed number in mostimplementations. Alternatively, in some implementations, a slice cancontain a varying integral number of CTBs with an approximately fixednumber of bits. The number of bits in a slice is referred to as theslice width. This invention proposes a new method of slice widthselection in the context of HEVC and compression/mosquito noisereduction.

As opposed to this characterization of slice width with regard to thegross video quality or noise, this invention uses a more nuancedapproach. In the context of this invention:

-   -   1. The edge map is first detected. As noted above, many        hardware-assisted primitives, including INTEL™ IPPs, are        available as non-limiting examples for this step or function,        and hence, the use of hardware-assisted primitives makes for low        complexity implementations, and the complexity of this step can        be expediently bounded. Example edge operators include Sobel,        Prewitt, Canny, Difference of Gaussian, Laplacian or other edge        operators.    -   2. Based on the edge map, the slice width may be chosen as        depicted in FIG. 20.    -   3. SAO filtering is selectively turned on or off in the loop        filter 322, based on the presence of an edge map or otherwise. A        measure of how may edge pixels are present in a specific region        can be used since, typically, regions would contain a few or        many edges (rather than none of completely edge region). More        advanced fuzzy classification methods can be used.

FIG. 20 illustrates Slice 0 (2000), Slice 1 (2002) and Slice 2 (2004)for the image from FIG. 15 using an adaptive slice width, wherein eachslice has a different width. Note that Slice 1 (2002) has contiguousblocks having significant edge content and hence this invention turnsSAO filtering on for only Slice 1 (2002) in this figure. This savesencoding complexity and compression efficiency for Slice 0 (2000) andSlice 2 (2004), while Slice 1 (2002) gets reduced mosquito noise aroundthe moving edges, by virtue of SAO filtering being turned on.

Formation of Slices

The edge map basically tags each block either as an edge-block ornon-edge block. FIG. 21 is an example edge map 2100, based on FIG. 20,that shows different arrangements and orientations of edge-blocks andnon-edge-blocks in a frame, wherein edge-blocks are labeled as “E”,while non-edge-blocks are labeled as “N”.

The edge (E) blocks within the boundary lines of 2102 (i.e., beginningat row 3, column 13 through row 7, column 14) satisfy the conditions toform an edge slice comprised of only edge blocks that are contiguous, aswell as more than the threshold required to form a slice. These blockscorrespond to Slice 1 in FIG. 20.

The edge (E) blocks outside the boundary lines of 2102 (i.e., beginningat row 1, column 1 through row 3, column 12, and beginning at row 8,column 1 through row 9, column 14) do not satisfy the conditions to forma complete slice that is comprised of only edge blocks, either becausethey are isolated, or in some cases contiguous, but too few in number,and hence below the threshold required to form an edge slice. Theseblocks correspond to Slice 0 (i.e., beginning at row 1, column 1 throughrow 3, column 12) and Slice 2 (i.e., beginning at row 8, column 1through row 9, column 14) in FIG. 20.

Slice 1 is formed first as a complete slice that is comprised only ofedge-blocks. After formation of Slice 1, the remaining blocks areaggregated into contiguous regions of blocks. A slice needs to have acontiguous region of blocks; hence, the aggregation yields two regionsfor slices (apart from Slice 1 already formed). Thus, the result isSlice 0 and Slice 2, respectively. Note that other slice formationmethods may be used for the non-edge slices (for example, based on aselected slice width for such non-edge slices).

The following describes a method for forming the exclusive slices withedge-blocks, according to one embodiment.

-   -   1. At the beginning of each frame, a toggle flag is initialized        to off and a counter is initialized to 0.    -   2. Pass 1—“Get Count Data”        -   Loop L—Each of the blocks in a frame is then traversed in a            raster scan.            -   Step A—“Count Blocks in Each Run”                -   As each block is encountered, if there is a                    transition from non-edge to edge, a toggle flag is                    turned on. Each edge-block increments a counter.                -   When there is a transition to a non-edge-block, the                    counter's value is saved to an indexed list of                    counter values associated with the contained blocks                    traversed in the run. (Each run of contiguous                    edge-blocks, that is, the run of edge-blocks                    encountered since the toggle flag was turned on, is                    saved as a separate indexed list entry of the                    counter's vale and associated blocks). The counter                    is then reset to 0 and the toggle flag is turned                    off.                -   The logic returns to step A, until all blocks of the                    frame are processed by Loop L.    -   3. Pass 2—“Form Slices”        -   A minimum threshold for blocks within a slice is selected,            e.g., N.        -   Each of the list entries in the indexed list of counter            values is examined. If the counter value is greater than N,            a slice is formed by the blocks associated with that list            entry. Note that N has an implication on the number of            slices in the frame and can be chosen accordingly. For            example, if the total number of blocks in the frame is B,            and it is decided that there should be no more than 5 slices            per frame, then N=B/S can be selected.        -   All the eligible runs which exceed N are formed as slices.            These are slices associated with edge-blocks for which SAO            filtering is turned on.        -   Each group of remaining blocks in between the eligible            slices are aggregated into a slice. Since there may be            multiple such groups in a frame, there could be multiple            such slices of non-edge-blocks, for which SAO filtering is            then turned off. Note that other slice formation methods may            apply for the non-edge slices (for example, based on a            selected slice width for such non-edge slices).

FIGS. 22A-22B are non-limiting examples of how some embodiments of theinvention would work on processors with multicore architectures.

FIG. 22A shows how the different slices are associated with differentfree cores available in an example timing diagram. Specifically, Slice 0is associated with core m, Slice 1 is associated with core m+1, andSlice 2 is associated with core m+2. Note that, in the case where thereare fewer cores than slices to be encoded, the number of blockscontained in each slice can be used to effectively load-balance thecores.

FIG. 22B shows a different timing diagram for a frame that hasadditional slices. Specifically, Slice 0 is associated with core m,Slice 1 is associated with core m+1, Slice 2 is associated with corem+2, Slice 3 is associated with core m, and Slice 4 is associated withcore m+2. In this example, cores m and m+2 handle multiple slices in thetime that core m+1 handles the largest slice, Slice 1. Effective taskscheduling mechanisms can ensure good parallelism among cores usingdifferent type of parallelization strategies.

Guidance on the SAO Filter to be Used

As noted above, the focus of this invention is to partition the videoframes into slices in order to turn SAO filtering on for selectedslices. The following steps can provide guidance on the SAO filter to beused.

-   -   1. The edges may show discontinuities or breaks, due to the        different strengths of edges. Short edges can be aggregated into        extended edges using some parametric techniques. Hough transform        can be used to parametrically identify line edges with different        angular orientations.    -   2. Using these detected and processed edges, a histogram of        edges is created for the edges with each block, that is, for        each coding tree block (CTB). Each histogram bin sums up the        strength of pixels with similar edge direction in the CTB using        the processed edges. The histogram bin with a maximum strength        indicates there is a strong edge along this direction in the CTB        and, hence, is considered as the likely edge offset type.    -   3. The SAO parameters for each CTB are comprised of SAO mode,        SAO type and four offsets. SAO type can be SAO_TYPE_EO_0, _90,        _135 and _45, and one band offset SAO_TYPE_BO. Since there are        four bins for edge offsets, the processed edges are clustered        into the closest bin, out of the said four bins.

Thus, based on the distribution of the edge direction histogram, onlyone Edge Offset class is chosen for further encoding process.

Experimental Results

Using edge-based processing, experimental results should thatimprovements were obtained using the invention. Specifically, FIGS. 23Aand 23B illustrate the difference in quality resulting from thisinvention, wherein FIG. 23B is of higher quality (i.e., less mosquitonoise around the edges) as compared to FIG. 23A. In order to ensure thatnon-edge block regions did not deteriorate, the PSNR-bitrate curves weremonitored and it was ensured there was no degradation due to theprocessing applied under this invention.

Hardware Environment

FIG. 24 is a diagram illustrating an exemplary computer system 2400 thatcould be used to implement elements of the present invention, includingsome or all of the elements of the codec system 200A. The exemplarycomputer system 2400 may also be used to encode the uncoded video 102according to the selected encoding parameters or to decode the codedvideo.

The computer 2402 comprises a general purpose hardware processor 2404Aand/or a special purpose hardware processor 2404B (hereinafteralternatively collectively referred to as processor 2404) and a memory2406, such as random access memory (RAM). The computer 2402 may becoupled to other devices, including input/output (I/O) devices such as akeyboard 2414, a mouse device 2416 and a printer 2428.

In one embodiment, the computer 2402 operates by the general purposeprocessor 2404A performing instructions defined by the computer program2410 under control of an operating system 2408. The computer program2410 and/or the operating system 2408 may be stored in the memory 2406and may interface with the user and/or other devices to accept input andcommands and, based on such input and commands and the instructionsdefined by the computer program 2410 and operating system 2408 toprovide output and results.

Output/results may be presented on the display 2422 or provided toanother device for presentation or further processing or action. In oneembodiment, the display 2422 comprises a liquid crystal display (LCD)having a plurality of separately addressable pixels formed by liquidcrystals. Each pixel of the display 2422 changes to an opaque ortranslucent state to form a part of the image on the display in responseto the data or information generated by the processor 2404 from theapplication of the instructions of the computer program 2410 and/oroperating system 2408 to the input and commands. Other display 2422types also include picture elements that change state in order to createthe image presented on the display 2422. The image may be providedthrough a graphical user interface (GUI) module 2418A. Although the GUImodule 2418A is depicted as a separate module, the instructionsperforming the GUI functions can be resident or distributed in theoperating system 2408, the computer program 2410, or implemented withspecial purpose memory and processors.

Some or all of the operations performed by the computer 2402 accordingto the computer program 2410 instructions may be implemented in aspecial purpose processor 2404B. In this embodiment, some or all of thecomputer program 2410 instructions may be implemented via firmwareinstructions stored in a read only memory (ROM), a programmable readonly memory (PROM) or flash memory within the special purpose processor2404B or in memory 2406. The special purpose processor 2404B may also behardwired through circuit design to perform some or all of theoperations to implement the present invention. Further, the specialpurpose processor 2404B may be a hybrid processor, which includesdedicated circuitry for performing a subset of functions, and othercircuits for performing more general functions such as responding tocomputer program instructions. In one embodiment, the special purposeprocessor is an application specific integrated circuit (ASIC).

The computer 2402 may also implement a compiler 2412 which allows anapplication program 2410 written in a programming language such asCOBOL, C, C++, FORTRAN, or other language to be translated intoprocessor 2404 readable code. After completion, the application orcomputer program 2410 accesses and manipulates data accepted from I/Odevices and stored in the memory 2406 of the computer 2402 using therelationships and logic that was generated using the compiler 2412.

The computer 2402 also optionally comprises an external communicationdevice such as a modem, satellite link, Ethernet card, or other devicefor accepting input from and providing output to other computers.

In one embodiment, instructions implementing the operating system 2408,the computer program 2410, and/or the compiler 2412 are tangiblyembodied in a computer-readable medium, e.g., data storage device 2420,which could include one or more fixed or removable data storage devices,such as a zip drive, floppy disc drive 2424, hard drive, CD-ROM drive,tape drive, or a flash drive. Further, the operating system 2408 and thecomputer program 2410 are comprised of computer program instructionswhich, when accessed, read and executed by the computer 2402, causes thecomputer 2402 to perform the steps necessary to implement and/or use thepresent invention or to load the program of instructions into a memory,thus creating a special purpose data structure causing the computer tooperate as a specially programmed computer executing the method stepsdescribed herein. Computer program 2410 and/or operating instructionsmay also be tangibly embodied in memory 2406 and/or data communicationsdevices 2430, thereby making a computer program product or article ofmanufacture according to the invention. As such, the terms “article ofmanufacture,” “program storage device” and “computer program product” or“computer readable storage device” as used herein are intended toencompass a computer program accessible from any computer readabledevice or media.

Of course, those skilled in the art will recognize that any combinationof the above components, or any number of different components,peripherals, and other devices, may be used with the computer 2402.

Although the term “computer” is referred to herein, it is understoodthat the computer may include portable devices such as cellphones,portable MP3 players, video game consoles, notebook computers, pocketcomputers, or any other device with suitable processing, communication,and input/output capability.

Processor Steps or Functions

FIG. 25 is a flowchart illustrating the steps or functions 2500performed by a processor, according to one embodiment. Specifically,these steps and functions are performed by the elements of the codecsystem 200A when encoding a frame of video.

Block 2502 represents a processor detecting an edge map comprised of oneor more edge-blocks in the frame. The edge map is detected by an edgeoperator. The edge map is detected by classification of pixels in theframe as edges or non-edges, and by classification of blocks asedge-blocks or non-edge-blocks based on the classification of thepixels. Specifically, the edge map is detected by a gradient ordifferences computation in a pixel domain of the frame, wherein a lowerthreshold and a higher threshold are used on the gradient or differencescomputation in order to generate the edge map, wherein the lowerthreshold and the higher threshold are used on a number of edge pixelsper individual block to classify the individual block as one of theedge-blocks or one of the non-edge-blocks, and wherein the lower andhigher threshold are scaled based on the individual block's size usedduring the encoding for decisions within the individual block.

Block 2504 represents a processor, when the edge-blocks are contiguous,forming at least one slice partition using the edge-blocks and encodingthe slice partition using a sample adaptive offset (SAO) filter. Theslice partition is formed with an adaptive slice width, and the sampleadaptive offset (SAO) filter is turned on or off during the encodingbased on whether the edge-blocks are being encoded.

Block 2506 represents a processor, when the edge-blocks are notcontiguous, performing edge-block processing around edges in the frameduring encoding of the edge-blocks. The edge-block processing involvesconfiguring one or more of: an intra block size, a transform block size,an inter prediction block size, a quantization parameter, candidatemodes for intra prediction, pyramid level for motion estimation, andfractional pixel motion estimation search.

Conclusion

This concludes the description of the preferred embodiments of thepresent invention. The foregoing description of the preferred embodimentof the invention has been presented for the purposes of illustration anddescription. It is not intended to be exhaustive or to limit theinvention to the precise form disclosed. Many modifications andvariations are possible in light of the above teaching.

It is intended that the scope of the invention be limited not by thisdetailed description, but rather by the claims appended hereto. Theabove specification, examples and data provide a complete description ofthe manufacture and use of the apparatus and method of the invention.Since many embodiments of the invention can be made without departingfrom the scope of the invention, the invention resides in the claimshereinafter appended.

What is claimed is:
 1. A method of encoding a frame of video,comprising: constructing an edge map that classifies blocks in the frameas being edge blocks or non-edge blocks, and using the edge map to format least one first slice partition having a size adjusted so as toenclose contiguous edge blocks, and to form at least one second slicepartition so as to enclose noncontiguous edge blocks; encoding blocks inthe at least one first slice partition using a sample adaptive offset(SAO) filter; and encoding blocks in the at least one second slicepartition by performing edge-block processing around edges in the secondslice partition, without applying a SAO filter to edge blocks in thesecond partition, wherein the SAO filter is disabled while encoding theat least one second slice partition.
 2. The method of claim 1, whereinthe slice partition is formed with an adaptive slice width.
 3. Themethod of claim 1, wherein the at least one slice partition comprises aplurality of slice partitions and the plurality of slice partitions areprocessed in parallel.
 4. The method of claim 1, wherein the edge map isdetected by an edge operator.
 5. The method of claim 1, wherein the edgemap is detected by classification of pixels in the frame as edges ornon-edges, and by classification of blocks as edge-blocks ornon-edge-blocks based on the classification of the pixels.
 6. The methodof claim 5, wherein the edge map is detected by a gradient ordifferences computation in a pixel domain of the frame.
 7. The method ofclaim 6, wherein a lower threshold and a higher threshold are used onthe gradient or differences computation in order to generate the edgemap.
 8. The method of claim 7, wherein the lower threshold and thehigher threshold are used on a number of edge pixels per individualblock to classify the individual block as one of the edge-blocks or oneof the non-edge-blocks.
 9. The method of claim 8, wherein the lower andhigher threshold are scaled based on the individual block's size usedduring the encoding for decisions within the individual block.
 10. Themethod of claim 1, wherein the edge-block processing involvesconfiguring one or more of: an intra block size, a transform block size,an inter prediction block size, a quantization parameter, candidatemodes for intra prediction, pyramid level for motion estimation, andfractional pixel motion estimation search.